feat(evals): passive/interactive agent eval framework over memory2 - #3411

Problem

Eval framework, supports both InteractiveEvals and PassiveEvals, @paul-nechifor old dimsim spatial memory eval replicated via the new framework here pretty clean:

BED = Vector3(-3.567, -1.332, 0.0)

go_to_bed = InteractiveEval(
    id="go_to_bed",
    inputs="go to the bed",
    score=lambda s: ramp((BED - s.streams.odom.last().data.position).length(), band=2.0),
    aggregate=final,          # or floor ("never left the zone") / mean / auc later
    blueprint="unitree-go2-agentic go2-memory",
    simulator="dimsim",
    scene="apartment",
)

Uses langchain openevals for VQA formatting so its compatible with all public benchmarks we'd want to run. Scoring is generic and done via lambda one-liners. Agent loop runs LLM queries with a light agent_loop in the EvalRunner -- kept minimal and using langchain client since just doing dumb QA.

Closes DIM-1392, DIM-1390

Solution

• new dimos/evals/ package. one word (evals) everywhere, no benchmark vs eval split
• two case types: PassiveEval (frozen mem2 recording, any replay window, non-autoregressive) and InteractiveEval (live sim/robot, actions mutate state, scored by sampling teh live recorder store)
• memory2 the source of truth for all eval input — context selectors return real Streams, no copies, no parallel data structres
• scoring is jsut lambdas over typed msg arithmetic (within, ramp, Vector3 subtraction etc), graded [0,1] credit not only pass/fail
• we use the langchain/openevals standard for vqa formatting (inputs / reference_outputs, evaluators wrapped not subclassed) so external benchmarks map on natively and we can test against them
• generic typing: PassiveEval[T] ties expected/parse/score togehter so mypy catches a mismatched case at suite definition time, not mid-run
• runnable via cli (dimos evals run ...), mcp (EvalModule skills so coding agents can iterate), and pytest
• --blind ablation built in — same suite with observations withheld, alredy caught two guessable MCQs
• MoondreamChat adapter so evals run keyless on any gpu box

defining evals

passive one-off over a replay — the whole thing is one literal:

from dimos.evals.scorers import first_number, within, yes_no, exact
from dimos.evals.types import PassiveEval, InteractiveEval, Suite

# numeric, graded: 1.0 exact, linear to 0 at band
distance = PassiveEval(
    id="distance_10min",
    inputs="How far have you traveled in the last 10 minutes, in meters?",
    expected=142.0,
    parse=first_number,
    score=within(15.0),
    context=(lambda s: s.streams.odom.range_time(0, 600),),   # real mem2 stream
    dataset="go2_hongkong_office",
)

# vqa over a 10-image window
person = PassiveEval(
    id="person_visible",
    inputs="Is a person visible in any of these images?",
    expected="yes",
    parse=yes_no,
    score=exact,
    context=(lambda s: s.streams.color_image.range_time(58, 61).limit(10),),
    dataset="go2_short",
)

interactive — the case names its environment, score is sampled from the live mem2 store every interval_s and reduced by aggregate:

from dimos.evals.scorers import final, ramp
from dimos.msgs.geometry_msgs.Vector3 import Vector3

BED = Vector3(-3.567, -1.332, 0.0)

go_to_bed = InteractiveEval(
    id="go_to_bed",
    inputs="go to the bed",
    score=lambda s: ramp((BED - s.streams.odom.last().data.position).length(), band=2.0),
    aggregate=final,          # or floor ("never left the zone") / mean / auc later
    blueprint="unitree-go2-agentic go2-memory",
    simulator="dimsim",
    scene="apartment",
)

running

from dimos.evals.runner import EvalRunner, summarize

results = EvalRunner().run(SUITE)                    # prod model config (gpt-5.6-luna)
results = EvalRunner(blind=True).run(SUITE)          # guessing ablation
results = EvalRunner(chat_model=MoondreamChat()).run(SUITE)  # keyless local
print(summarize(results))

or dimos evals run dimos.evals.suites.go2_smoke --blind. every run writes results.jsonl + summary.json + per-case transcripts to ~/.local/state/dimos/evals/run-*/.

first real numbers

luna sighted: examples 1.00 / smoke 1.00 / vqa 0.86, blind = refusals. interactive rig scored 0.856 on a scripted go-to-bed in dimsim. the full agentic go-to-bed scores 0.0 right now — unitree-go2-agentic publishes no /odom in dimsim on current main (upstream test_dimsim_spatial_memory fails teh same way, suspect the control coordinator refactor). separate ticket coming.

Breaking Changes

None

How to Test

uv run pytest dimos/evals dimos/codebase_checks -q          # offline unit + wiring tests, no keys
uv run pytest dimos/evals/test_smoke.py -m self_hosted      # live model smoke (needs OPENAI_API_KEY + lfs data)
dimos evals run dimos.evals.suites.examples                 # 2 doc cases against go2_short
dimos evals run dimos.evals.suites.go2_smoke --blind        # guessing ablation

interactive (needs deno + display): dimos evals run dimos.evals.suites.dimsim_house --live-db recording_go2.db

Contributor License Agreement

• I have read and approved the CLA